cookbook(action): add RoboCasa mobile-manipulation post-training recipe - #318
cookbook(action): add RoboCasa mobile-manipulation post-training recipe#318hyzhou-nv wants to merge 1 commit into
Conversation
|
Thanks! In general it LGTM, but can we keep this recipe focused on training? So files are organized in the same way of other sft recipes. Maybe we can move the eval/dataset related scripts to cosmos-framework and add the usage guidance in the readme? |
0a8d25e to
2161358
Compare
Thanks for the suggestion! Both eval and dataset scripts are moved to cosmos-framework repo. |
|
Seems a lot of changed unrelated to robocasa is added? |
2161358 to
9079a60
Compare
Adds a RoboCasa recipe to the action finetune cookbook, organized like the DROID and LIBERO ones: a training launcher, the Nano SFT TOML, and README coverage. Evaluation and dataset conversion live in cosmos-framework, so the recipe folder holds training files only. RoboCasa is mobile manipulation, so the base is not fixed. The recipe uses a 15-D raw action contract that passes the native base command through unchanged, an agentview_left | eye_in_hand composite at full resolution, and EEF proprioception as a clean condition token. `TASK_SET` selects the split (18 target/atomic, 65 pretrain/atomic, or all 300); the names resolve to the task tuples in `robocasa_lerobot_dataset.py`, which stays the single source of truth. Conversion has to run before training rather than being optional setup: RoboCasa publishes LeRobot v2.1 and the pinned lerobot is v3.0 only and rejects that layout. The README documents the ordering and where the converter and evaluator live. Requires the matching cosmos-framework change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9079a60 to
3f3a82e
Compare
Sorry about that. I mistakenly created the PR from an outdated commit. I've rebased and force-pushed. |
fwd4
left a comment
fwd4
left a comment
There was a problem hiding this comment.
Re-review (requesting changes):
The unrelated-file scope problem is fixed, but the cookbook documentation still does not describe a runnable closed-loop evaluation command with the companion framework implementation:
-
[P1] The documented 15-D contract is missing the server-side width configuration. This README asks the evaluator to use
--use-state --use-base-action --base-encoding raw(15-D), but it does not show the policy-server command or--raw-action-dim 15. The companionaction_policy_server_robocasadefaults to 7 when the width is not inferred, so the evaluator sends a 15-D state token and the first/predictrequest fails. Please add exact two-process commands and the explicit width flag, or fix the server to infer the effective width. -
[P2] Keep the two repositories’ contract in one source of truth. The training TOML, launcher, README, evaluator, and server currently repeat the action width and camera/state settings. A small documented smoke test using the exact cookbook commands would catch drift between these two PRs before merge.
Until the server/evaluator handshake is fixed and exercised, the RoboCasa recipe is not runnable as documented.
2 participants
Adds a complete RoboCasa recipe to the action finetune cookbook, following the DROID/LIBERO layout: dataset conversion, training launcher, closed-loop evaluation, and the Nano SFT TOML.
RoboCasa publishes LeRobot v2.1 while the pinned lerobot is v3.0-only, so a one-time conversion step is included; it works on copies and is idempotent.
Evaluation follows the official protocol -- a policy server per GPU with the simulator client in a separate venv, since robosuite and cosmos-framework cannot share one environment.
Requires the matching cosmos-framework change that registers the RoboCasa dataset and embodiment.